The Pitfalls of Sample Selection: A Case Study on Lung Nodule Classification

Using publicly available data to determine the performance of methodological contributions is important as it facilitates reproducibility and allows scrutiny published results. In lung nodule classification, for example, many works report results on LIDC dataset. theory, this should allow a direct comparison proposed methods assess impact individual contributions. When analyzing seven recent works, however, we find that each employs different selection process, leading largely varying total number samples ratios between benign malignant cases. As subset will have characteristics with difficulty thus not always possible, nor fair. We study particular effect truthing when aggregating labels from multiple experts. show specific choices can severe distribution where may be possible achieve superior one sample but another. While further improve state-of-the-art selection, also more challenging same database, advanced models underperform respect very simple baseline methods, highlighting selected play an even role than model architecture. This raises concerns about validity claimed believe community aware these pitfalls make recommendations how avoided in future work.